Back to Main Menu

Process: Retrieve Assetic Data with Spatial ETL Tools

Introduction

Asset attributes may be first created from the GIS as part of the asset creation workflow, particularly if a developer or other organisation has created the asset and is providing the 'as-constructed' assets in spatial format with asset attributes. However, after creation the asset's attributes will change over time within the Assetic environment. As Assetic is the source of truth for attributes, a GIS may need to retrieve and reference the current attribute values contained within Assetic.

 

A common approach to this is by exporting and making use of a local extract of Assetic data as outlined in the article: Creating a Local Backup of Assetic Data (Assetic Python SDK).

 

However, the Assetic REST API or OData endpoints can also be utilized to connect with and obtain attribute data where the capability to compile and execute a HTTPS request is available.

 

The Assetic asset ID held within the GIS system can be used as the key to join the Assetic attribute data to the spatial features for the purpose of merging the up-to-date attribute data with the GIS spatial information.

 

Due to the condition that any request to the Assetic REST API or OData endpoints are HTTPS requests, this process needs to include the authentication for the Assetic environment within the request header. Authentication is via the "Basic" HTTP Authentication scheme, where a Username and Password pair (encoded using base64) is supplied within an 'Authorization' header. 

 

The Rest API Introduction article contains further information on authentication, including how to generate an API token to use in place of the standard password for integrations.

FME Workbench and Data Interoperability

OData

Create OData Reader

You may need to right click and ‘Save As’ to download the example Workspace.

 

The FME OData Reader produces an FME feature for each entry in the specified OData feed. Each property (i.e field) on a feed entry can be utilized as an attribute on the resultant FME Feature.

 

The FME OData reader currently supports version 2.0 through 4.0 of the protocol specification.

 
Note: The Assetic OData version is v4.0

 

Within the Reader parameters, the' Source OData Service' parameter should contain the service root URI of the Assetic environment's OData service, which returns a list of feeds available for consumption. 

 

The Authentication should be set to Basic, and the Username and Password authentication values for the integration user are supplied.

 

The Assetic OData endpoints support a maximum of 10,000 records returned. This upper limit should be specified within the reader for both the 'Features Per Request' and 'Max Features to Read' parameters to avoid errors. If not specified, FME will attempt to iteratively request using the $skip operator, resulting in an error.

 
The Assetic OData endpoint will only ever return the first 10,000 results regardless of $top and $skip. The $skip parameter cannot be used to skip the first 10,000 records to expose the next 10,000 records.

 

The image below shows an example configuration for the FME OData reader parameters:

Configure Feature Type Parameters

After the OData reader has been created, the transformer's parameters can be modified to specify the operations. The available query options in the transformer are: 'Order By', 'Filter', and 'Expand'.

 

The FME OData Reader does not provide a 'Select' ($select) query parameter on the assumption that by omitting '$select', all fields will be returned by the endpoint. This assumption is not supported by Assetic, and a '$select' query MUST be provided with the desired fields specified in order for results to be returned.

 

As a 'Select' query option is not present in FME, to achieve this requirement a work-around is to specify the '$select' query options within one of the other existing query options (e.g 'Sort'), as an appended statement.

 

This is done by using the '&' character to separate and then include the '$select' query, as seen in the below example:

 

Note: The OData Endpoint article contains additional information about the available Assetic OData endpoints, and the supported operations

Configure User Attributes

After a 'Select' ($select) query has been specified in the OData parameters, the 'User Attributes' tab can be modified to only make use of the fields specified in the '$select' statement. Selecting only the required fields significantly reduces the overhead of the process, as needless OData properties are ignored during run time.

 

By default, FME will expose attribute definitions for all properties returned by the OData entity's feed specification. For the Assetic 'assets' entity, this can be a substantial total number of fields (approximately 4000 fields across all asset categories) and will significantly slow the runtime of the FME process. Instead, only required fields should be exposed.

 

To do this, within the 'User Attributes' tab of the OData transformer, select the 'Exposed Attributes' toggle for the "Attributes to Read" section. Then, select the checkboxes to only expose the fields that match the 'Select' query statement.

 

This ensures that only the relevant fields specified within the '$select' query are parsed by the OData reader when the endpoint is queried, and all other fields that are not required are ignored.

REST API

Requests Using HTTPCaller

You may need to right click and ‘Save As’ to download the example Workspace.

 

Using a FME HTTPCaller transformer allows the ability to perform a HTTP request to a specified URL from within a FME workflow. The transformer accesses a specified URL via HTTP or HTTPS, using a variety of HTTP methods including those supported by the Assetic REST API (i.e. GET, PUT, and POST).

 

An image of an example configuration for the HTTPCaller parameters can be found below: 

 

This example shows the 'Request URL' is specified for retrieving a collection of asset information from the "GET /api/v2/assets" endpoint. The "Query String Parameter" section allows for input for any API request parameters required such as Page, Page Size, Filters, Attributes, or Sorting. The request parameters available for use during a request varies depending on the API endpoint.

 

The use of the REST API parameters is outlined in the Search Filters and Pagination article.

Handling REST API Responses

Responses from the Assetic REST API endpoints are in a JSON structure. FME has various JSON transformers available, such as 'JSONExtractor' and 'JSONFragmenter', that can be used to transform the JSON response into individual features with exposed attributes based on the API response.

 

The below example workflow process shows a sample approach to handling a JSON response from an Assetic API request:

 

In this example, the output of API response is first extracted using the 'JSONExtractor' transformer to split the properties of the JSON response into individual attributes, which will be further fragmented in a future step:

 

The 'JSONFragmenter' transformer then allows for the extracted JSON array to be flattened into individual features:

 

In this example, the array of results is contained within the previously extracted 'ResourceList' attribute and has been selected as the 'Input Source' to be fragmented.

 

Any fields contained within the JSON response object that are required to be exposed for further handling within the FME process must be manually specified within the "Attributes to Expose" parameter.

 

After exposing, each record from the API response will be contained to its own feature row and can be used for any future transformations:

 

Subsequent fragmentation or extraction transformations may be required, depending on the output JSON structure, as API responses may have properties that hold a further level of JSON object data (e.g. asset attributes) that need to be flattened to attributes.